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Who am I? 


ChangKyu Song 

Head of Live Infra Technology Department, Nexon 

• 1999, developed HanStar, a Korean Localization Utility for Starcraft 

• 1999, Gabriel Knight 3 Korean Localization 

• 2001 , Worms World Party Korean Localization 

• 2002, Crazy Arcade BnB (Arcade) Programmer 

• 2002-2003, Dizzy Pang (Puzzle Arcade) Lead Programmer 

• 2004-2006, Big Shot (2D Shooter) Lead Programmer 

• 2006-2010, Bubble Fighter (Third-person Shooter) Lead Programmer 

• 201 0-201 1 , Mabinogi 2 (3D MMORPG) Programmer 

• 2011-2014, Dungeon & Fighter (2D Action MORPG) Technical Director 

• 201 4-, Head of Live Infra Technology Department, Nexon 
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About Nexon 

• A Korean Publisher 
Running many games for a long time 
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About Nexon 

• A Korean Publisher 

Running many games for a long time 

• >30 PC titles 

• Dungeon & Fighter, Maple Story, 
Bubble Fighter, .. 
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About Nexon 

• A Korean Publisher 

Running many games for a long time 

• >30 PC titles 

• Dungeon & Fighter, Maple Story, 
Bubble Fighter, .. 

• >20 mobile games 

• DomiNations, HIT, Legacy Quest .. 
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About Nexon 

• 201 5's full year revenue: 

$1.67 billion 

(190.3 billion yen) 


Last week Nexon announced its financial results for the fourth quarter and fiscal year 
ended December 31 , 201 5 with financial growth across the board. The full year revenue 
for Nexon came to 190.3 billion yen {USD1.67 billion), a 1 0 per cent jump on 
2014. Meanwhile, operating income grew by over a third (37 per cent) to reach 62.3 
billion yen (USD 0.51 billion). Nexon's Clash of Clans-esque strategy title DomlNations 
was among the new releases helping to drive revenue, as the game's popularity surged 
following its release in April. 

Q4 alone delivered a seven per cent rise versus the previous year, with 'higher than 
expected' Dungeon Fighter and HIT sales boosting revenue to 45.8 billion yen (USD 0.37 
billion). 

"Nexon's solid performance throughout 2015 
was driven by our relentless focus on the 
quality of our live and new games and the 
expansion of world class partnerships. Our 
full-year revenues, operating income and net 
income all grew year-over-year, reflecting the 
positive response from players of our high- 
quality titles," said president and CEO Owen 



Nexon CEO Mahoney 
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Fighting against Legacy - 
Running game for a Long Time 

Finding hotspot in source code by change analysis 
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Working on Dungeon & Fighter 


• Joined and worked as Global Development Lead and Technical Director in 201 1 

• One of most successful game in Nexon 

(*Neople is a subsidiary company of Nexon) 


• Launched On Aug 2005 (10 years old) 

• More than 3 million concurrent users in China 

• More than $1 .0 billion revenue in 201 5 

• Still one of most top-grossing MMOs in 201 6 



Digital PC 


1. "League of Legends" (Tencent/Riot Games, $1,628) 


2. "Crossfire" (SmileGate, $1,110) 


3. "Dungeon Fighter Online" (Neople, $1,052) 


4. "World of Warcraft" (Activision, $814) 
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When running live service for a long time. 

Code size grows, team doesn't scale 





2282 .cpp files 
2935 .h files 
5855 classes 


^\\e 
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Game grew too big 

• As in 201 4 (After 9 years of service): 

• >2,000 skills 

• >4,500 quests 

• >10,000 C++ source files 

• >50,000 equipment items 

• >200,000 animations 

• >5,000,000 lines of code 

• >10,000,000 images 

• The biggest and fastest-paced project ever experienced 
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Needed to pay off Technical Debt 


• Dev team was suffering from technical debt 

• Adding very small content was very costly 

• Needed to pay off technical debt 


Finding the realistic, efficient way 


• >1 0,000 source code files 

• Most code smelled bad 

• Rather than understanding gigantic size of code 
and improving hand by hand 

• Decided to take advantage of automation 
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Doing some Automation 

• Introduced Build Server, Data Validation Test and Static Analysis 

• Build Server, Data Validation helped a lot 

• Improving notification made it much better 

• Static Analysis didn't help much 

• Fixed thousands of warnings over weeks with many people 

• Valuable but high cost, little gain 

• Need to focus on important thing/area 

• Added basic things like crash reporter, 

• "What is important?" 

• Where is the important area? - each developer said different area 


iiik 
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Finding what's important for Software quality 

• Separating frequently-changing area and less-frequently 
changing area is important 


Example case 

Frequently-changing area 

Less frequently changing area 

Software Engineering 

Interface 

Implementation 

STL 

Algorithms 

Data Structure 

Template/Generic 

Logic 

Data Type 

Game Engine / Framework / Library 

Library Code 

Logic Code 

Data-driven Development 

Data 

Code 
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Finding hotspot: Principle of locality 

• If particular source location is changed, it is likely to 
change again in the near future 

Types of locality^ [ edit ] 

There are several different types of locality of reference: 

Temporal locality 

If at one point in time a particular memory location is referenced, then it is likely that the same location will be referenced again in the near 
future. There is a temporal proximity between the adjacent references to the same memory location. In this case it is common to make efforts to 
store a copy of the referenced data in special memory storage^ which can be accessed faster. Temporal locality is a special case of the spatial 
locality, namely when the prospective location is identical to the present location* 

Spatial locality 

If a particular memory location is referenced at a particular time, then it is likely that nearby memory locations will be referenced in the near 
future, in this case it is common to attempt to guess the size and shape of the area around the current reference for which it is worthwhile to 
prepare faster access. 
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Analyzing change of code over time 

• Not just doing static analysis which treats code as static data, 

• But also tried to treat code as changing data and 
analyze code changes in VCS over time 


Static analysis approach: 


f >1 

Static data 

V J 


Expanded approach: 
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First step: Simple Analysis 

• How many files have changed and 
how often has each changed? 


Wrote 300 lines of python script to analyze 

change of source code 


262 

263 

264 

265 

266 

267 

268 

269 

270 

271 

272 

273 

274 

275 

276 

277 

278 

279 

280 
281 
282 

283 

284 

285 

286 

287 

288 

289 

290 

291 

292 

293 

294 

295 

296 

297 

298 

299 

300 

301 

302 

303 

304 

305 
■36 
337 
308 

310 

311 

312 


for i in xrange(len(msgs)) : 
msg = msgs[i] 

print "processing *d / %6“ * (i+1, len(msgs)) 
logmsg = msg. message. decode("utf-8") 

logFlags = set() 

if hasKeywords( logmsg, ModifyKeywords) : 
logFlags . add( "modify" ) 

if hasKeywords(logmsg, BugfixKeywords, BugfixExcludeKeywords) : 
logFlags . add( "bugfix" ) 
if hasKeywords( logmsg, RevertKeywords) : 
logFlags . add ("revert ") 

# 500?H 01^^21 

if len(msg.changed_paths) > 500: 
continue 

for chg in msg. changed_paths : 
if chg. action != 
continue 

st = time.localtime(msg["date"]) 
unitTime = time.mktime(( 
st.tm_year, 

St .tm_mon, 

st.tm_mday - st.tm_wday, 

0 , 0 , 0 , 0 , 0 , 0 )) 

fullpath = chg. path. decode("utf-8") 

path = fullpath 

if path. startswith(repopath) : 

path = path[len(repopath) : ] 

if not path.endswithC'.cpp"): 
continue 

url = repobase + fullpath 
url = url.replaceC ", "*20") 

try: 

procOne(url, msg. revision. number, unitTime, logFlags) 
except Exception as e: 

fe = openCexception.log", "at") 

fe.write("*d:*s\n" % (msg. revision. number, fullpath. encode("cp949"))) 

fe.write("*s\n" * e. message) 

fe.closeO 


print "writing.." 
f = openC'FuncHistory . pkl", "wb") 
cPickle.dump((tbl, colMin, colMax), f, 2) 
f .closeO 
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/IDAR/CNWarRoom Ma ".aqe'.cpp 










































oo / 










































ooo 

/RDAR/CNWarRasm Ma -.aq^'.-. 
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/RDAR,''CN WarRoom Ms3 x 0 XOD 
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More readable result 
Zoomed out (group by week) 

& Sorted by recent change, frequency of change 






Bird's-eye view 
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/ 1 1 J /RD ARCha ra-ct&r/CN Pri estc p p 

3 IS 

/1 14 /RDARChara-cter/CNThielcpp 

327 

/il^ /RDAR^lRDAttiveObjecth 

330 

/lib /RD AR/Cha ra'Cter/CN ATFi g hter.c pp 

336 

/II/ '/RDAR/[RDPopupWindow.cpp 

341 

/liy /RDAR/[RDCharatter.h 

390 

/iiy 

/RD AR/CN RD Vi rtua iCha ra^ter.c pp 

412 

/l^U 

/RD AR/Cha rat ter/CNS wo rd ma n.t pp 

459 

nzL 

/RDAR/Cha ratter/CN G u n n er.t pp 

464 

/12Z 

/RD ARf Ga m e€o rein itc pp 

4S6 

/i^T 

/RDAR/Cha ratter/CN Fi g hter.c pp 

502 

/1^4 

/RD AR^'Po p u pWi n do w/CN RDItemlnfo Wi n d o w.t pp 
/RD AR/IRDC 0 1 1 i s io n 0 bj e<tc pp 

507 

510 

^mw 

/RD AR/CN RD Ud pM o d u 1 e.t pp 

525 

rnr 

/RDAR/Cha ratter/CN Ma g e.t pp 

537 

/i^y 

/RD AR/[RD Acti veO bj ectc p p 

LTi 

'd- 

LTl 

/i^y 

/RD AR/CN RDInterfat eMa na g er.c p p 

6S6 

/iJU 

/RD AR/CN RD E q u i p m e ntc p p 

6S3 

/idi 

/RD AR/CN RD Sta g e.c p p 

75S 

/id!2 

/RD AR/[RD Mo n ster.c pp 

779 

/idJ 

/RD ARf Ga m eC o reS u b.t p p 

SIS 

/li4 

/RD AR/CN RDC ha tti n g.c p p 

S31 

/11b 

/RD AR/RDC 0 m m o n.t pp 

907 

/11b 

/RD ARy'RD AR_vt pro] 

967 

/ll/ 

/RD AR/CN RD Ea s elnterfat e.t pp 

995 

/ liy 

/RD AR/IRDC ha ratter.t pp 

1077 

/iiy 

/RD AR/’Ga m eCo reN etwo rk_CM D.t p p 

1107 

/140 

/N e 0 pi eMa i n/d nf.str 

1759 

# 14A 1 /RD AR/Ga m eCo re N ettvo rk_N OTLc p p 

ISSS 


Source files that are changing often 

= Hotspots 

Highly efficient when improved 







200S-11-2S 2011-OI-2S 
ti MOS-11-3 
n 20os-u>:i 

17 20Qi-07.a »U-M-2S 

20i»^-o« ioi: 
200S.ll.2i 2011.Ot.2S 
200S.ll.2i 2011.OI.2S 
2 t 200S.li.2i 2011.O4.2S 
200S.11-2I 2011.O4.2S 
200S.ll.2t 20U.04.2S 
20as.ll.2i 2011.O4.2S 
l.2i 2011.04.2S 

200S.U.2i 2011.O4.2S 
L.M 20U.04-25 
200S-ll-2i 2011.04.2S 
200S.ll.2i 2011.04.2S 
L.2I 20U.O4.2S 

2007.02.11 2011.04.25 

2001.02.11 20U.O4.2S 

r-U 2011.O4.2S 
200S.112i 2011.04.2S 

200S.ll.2i 2011.04.2S 

1.21 2011.04.2S 
200S.ll.2i 2011.O4.2S 
200S.U.2i 2011.O4.2S 
20QS.ll.2i 2011.04.25 
20QS.112i 2011.04.25 
200S-U.2i 20U.O4.2S 
200S.ll.2i 2011.04.2S 
200S-11.2i 2011.04.25 

1.21 2011.04.25 
200S.ll.2i 2011.04.25 
200S.ll.2t 2DU.04.2S 
200S.tl.2i 2011.04.25 

l.» 2011.04.2S 
L.2i 2011.04.25 
200S.ll.2i 20U.04.2S 
200S.ll.2t 2011.04.25 
200S.112i 2011.04.25 
200S.ll.2i 20U.O4.2S 
200i.02.20 2011.04.2S 
200i4)l.li 2011.04.2S 
N20 2011.04.25 




File 

chargt 

min 

max 

/WTL80/i nc 1 ude/atl resce.h 

1 

2009-03-16 

2009-03-16 

/WTL80/i nc 1 ud e/atl sc rl.h 

1 

2009-03-16 

2009-03-16 

AVTL80/i nc 1 ude/atl s pi ith 

1 

2009-03-16 

2009-03-16 

/WTLSO/include/atItheme.h 

1 

2009-03-16 

2009-03-16 

/WTLSO/include/atluser.h 

1 

2009-03-16 

2009-03-16 

/WTL80/i nc 1 ude/atl wince.h 

1 

2009-03-16 

2009-03-16 

/WTL80/i nc 1 ude/atl winx-h 

1 

2009-03-16 

2009-03-16 

/WTL80/readme-htm 

1 

2009-03-16 

2009-03-16 

/Tools/DNFSpline 

1 

2009-03-23 

2009-03-23 

/T ools/DNFSpline/DNFSpline.sIn 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/DNFSpline.cpp 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/DNFSpline.h 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/DNFSpline.vcproJ 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/DNFSpline_2005.rc 

1 

2009-03-23 

2009-03-23 

/Too Is/D N FS pi i ne/D N FS pi i ne/Read Me.txt 

1 

2009-03-23 

2009-03-23 

/Too 1 S/D N FS pi i ne/D N FS pi i ne/res 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/res/DNFSpline.ico 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/res/DNFSpline.manifest 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/res/DNFSpline.rc2 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/res/DNFSplineDocico 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/res/Toolbar.bmp 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/stdaficcpp 

1 

2009-03-23 

2009-03-23 

/Tools/DNFSpline/DNFSpline/stdafich 

1 

2009-03-23 

2009-03-23 

/DNRnGameTools/lib 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/lib/ticpp.lib 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/lib/ticppd.lib 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/TiCPP.vcproj 

1 

2009-05-11 

2009-05-11 

/D N RnGa meToo 1 s/ticpp/TinyXM L-*- .neb 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/TinyXML--.sln 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/TinyXML-^*.suo 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/build_instructions.bct 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/changes.txt 

1 

2009-05-11 

2009-05-11 

/DNRnGameToolsAicpp/dox 

1 

2009-05-11 

2009-05-11 

/DNRnGameToolsAicpp/premake.exe 

1 

2009-05-11 

2009-05-11 

/DNRnGameToolsAicpp/premake.lua 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/premake4.lua 

1 

2009-05-11 

2009-05-11 

/D N RnGa meToo 1 s/tiepp/read me.txt 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/ticpp.cpp 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/ticpp.lua 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/ticpp4.lua 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticppAicpprc-h 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/tinystr.cpp 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticppAinystr.h 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/tinyxml.cpp 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/tinyxmLh 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/tinyxmlerror.cpp 

1 

2009-05-11 

2009-05-11 

/DNRnGameTools/ticpp/tinyxmlparser.cpp 

1 

2009-05-11 

2009-05-11 

/Tools/EquipmentImagePack 

1 

2009-06-22 

2009-06-22 

/Tools/EquipmentlmagePack/EquipmentlmagePacIccpp 

1 

2009-06-22 

2009-06-22 

/Tools/EquipmentImagePack/EquipmentImagePacIcncb 

1 

2009-06-22 

2009-06-22 

/Tools/EquipmentlmagePack/EquipmentlmagePack-sIn 

1 

2009-06-22 

2009-06-22 


/Tools/EquipmentInagePack/EquipmentImagePacIcsuo 1 2009-06-22 2009-06-22 
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For source files that change rarely 
we can consider separating them from the 
main project and reduce build time 


WTL (Windows Template Library) 
DNFSpline: 

Tinyxml 

EquipmentImagePack 


Each name looks a lot like library code 
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> 


1 1 >Comp i I i ng . . . 

1 1 >prof i I er . cpp 
1 1 >Comp i I i ng . . . 

1 1 >TEA . cpp 
1 1 >Comp i I i ng . . . 

1 1 >sha256 . cpp 
1 1 >Comp i I i ng . . . 

1 1 >SHA . cpp 
1 1 >Comp i I i ng . . . 

1 1 >R i j ndae I . cpp 
1 1 >Comp i I i ng . . . 

1 1 >Method . cpp 
1 1 >Comp i I i ng . . . 

1 l>CNRDUIEnchantTool 
1 1 >Comp i I i ng . . . 

1 1 >S I angF i I terSub . cpp 
1 1 >S I angF i Iter . cpp 
1 1 >Comp i I i ng . . . 

1 1 >RDChamp i onConst . cpp 
1 1 >Comp i I i ng resources 
1 1 >M i crosof t 
1 1 >Copyr i ght 
1 1 >L i nk i ng . . 

1 1 >Bu i I d Tim 
11>RDAR - 0 er 
========== Rebu i I d 


1 >wave . c 
1 >CWebPage . c 
1 >Comp i I i ng . . . 
1 >prof i I er . cpp 
1 >Comp i I i ng . . . 
1 >TEA . cpp 
1 >Comp i IJ ng . 

1 >sha256i 
1 >Comp i I r 


cpp 



53% decreased 


(x2.1 improvment) 



1 l>SlangF 
1 l>SlangF 


(R) 

(C) 


Windows (R) Resource Compi 
Microsoft Corporation. Al 


:32: 54 


73 warning(s) 
All: 11 succeeded. 


0 fai 


?rSub . cpp 
fter . cpp 
1 1 >Comp i I i ng . . . 

1 1 >RDChamp i onConst . cpp 
1 1 >Comp i I i ng resources 
1 1 >M i crosof t 
1 1 >Copyr i ght 
1 1 >L i nk i ng . 

1 1>Bui Id Ti 

11>RDAR - 0 ?PTui 73 warning(s) 
========== Rebuild All: 11 succeeded. 0 fai 


(R) 

(C) 


Windows (R) Resource Compi 
Microsoft Corporation. Al 


15: 37 


Before 


After 


Applying unity build technique for the half of less-frequently-changing 

code made compiling twice as fast. 
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cnn^e.h 

cngil^er.h 

onib • 

cnguni^ 9 bull«t.h cwivMwb 


cnp^st.h 


h c»kw<«»i(a««Ktf i< cncl«i(j^Mar.li 

cnsw(10man.h 

y, fe CfirdlkllMo^ 

■hii^rlai li C««O0e««rtO0»^ll«*wb H 

, -,t,M ■ cn uw-^»i««» cnrdskil^nager.h 


K cnrdeqdlbment.h 

itemf4l^ory.h 


iv^Obrywindow.h . cnrdca^^op«».h 

cnrdcai^hop.h 


cnfiJPter.h 


cnma|V>^ct.h 

cnatf^iter.h 


cnrdnew^^hthop.h 

cnrdvirtui^aracter.h 

irdaicH^felcter.h cncr«atur4i|uipment.h 

BiobiKicion h cnrdbas^erface h 


• irdchl^lcterh 


cnrdvirtu^ feature. h 


Visualizing dependency graph 
to check innpacts of files when changed 

Not good if files with big impact 
change frequently 


irdcr4||ure.h 


eiw>«e^k 
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Considering about Change Impact 


• Change Impact = 

Change Frequency * Impact (=Reverse-dependencies including itself) 


GDC 
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A 

B 



D 

E 

F 

G 

H 

I 

J 

K 

L 

1 

Ffe 

Impact S're 

Impact Count 

02/28 

03/07" 

03/14 

03/21 

03/28 

04/04 

04/11 

04/18 

04/25 

38 

cnscnptioader.h 

443 

2 



0 

0 

220 

0 

0 

0 

0 

223 

39 

cnwarroom.h 

508 

2 



253 

0 

255 

0 

0 

0 

0 

0 

40 

cngamesocketh 

582 

2 



0 

0 

285 

0 

0 

0 

0 

297 

41 

c n rd'rtemi nfowi ndow. h 

585 

7 


0 

162 

0 

82 

0 

0 

0 

255 

86 

42 

cnrdudpmodule.h 

596 

2 


0 

0 

596 

0 

0 

0 

0 

0 

0 

43 

udpcore.h 

618 

2 



0 

307 

311 

0 

0 

0 

0 

0 

44 

icreature.h 

800 

4 



0 

0 

0 

0 

0 

0 

598 

202 

45 

c n rdq uestmod ule, h 

875 

4 



0 

0 

0 

0 

436 

218 

0 

221 

4€ 

c n rdv rtuak ha racter.h 

925 

6 



0 

0 

0 

0 

154 

462 

309 

0 

47 

cnrdmovementjnith 

1145 

3 


380 

0 

0 

0 

0 

0 

765 

0 

0 

48 

c n rd Lila utomat oncontrotoader.h 

1207 

8 


125 

0 

258 

266 

136 

139 

0 

0 

283 

49 

c n wa rroom mod jle.h 

1218 

5 


0 

0 

0 

244 

244 

243 

243 

244 

0 

50 

c n rd uia utomat'onsystem.h 

1248 

7 


0 

0 

0 

132 

266 

270 

0 

0 

580 

51 

udpcommondefine.h 

1265 

4 


0 

0 

309 

632 

0 

0 

324 

0 

0 

52 

cnrdchattng.h 

1310 

5 


0 

0 

0 

0 

257 

521 

0 

0 

532 

53 

cnusermanager.h 

1691 

3 


0 

0 

0 

557 

1134 

0 

0 

0 

0 

54 

corditemmanagerh 

2003 

3 


656 

0 

0 

663 

0 

0 

0 

0 

684 

55 

cnrdstage.h 

2182 

2 


0 

0 

0 

1089 

1093 

0 

0 

0 

0 

56 

c n rdobjectma nager.h 

2243 

3 


0 

0 

0 

742 

0 

0 

0 

749 

752 

57 

c n rd; nterfacema nager.h 

3135 

6 


0 

0 

516 

1040 

1051 

528 

e 

0 

0 

58 

c n rd base' nterface. h 

3223 

18 


171 

344 

0 

176 

712 

905 

181 

734 

0 

59 

gamecore.h 

3953 

11 


344 

693 

694 

1059 

0 

0 

722 

0 

441 

60 ; 

cnaJmonster.h 

3979 

4 


0 

0 

979 

0 

0 

2000 

1000 

0 

0 

61 

rdsblstate-h 

4017 

8 

1 

0 

0 

0 

1979 

2023 

0 

15 

0 

0 

62 

cnrdabteobj’ecth 

4949 

5 


0 

0 

1965 

0 

1987 

997 

0 

0 

0 

63 

c nsklstatema nager.h 

4982 

8 

1 

0 

0 

0 

3953 

1016 

13 

0 

0 

0 

64 1 

cnrdsldlh 

5030 

5 


0 

0 

989 

1003 

1007 

1010 

0 

1021 

0 

65 

rda c ha racter.h 

5208 

12 


0 

0 

856 

0 

870 

435 

870 

0 

2177 

66 

dfggmglog.h 

5305 

2 


0 

0 

0 

0 

0 

5305 

0 

0 

0 

67 

'mojse.h 

5322 

2 


0 

0 

0 

0 

0 

5322 

0 

0 

0 

68 

Tmage.h 

5463 

2 


0 

0 

2714 

0 

2749 

0 

0 

0 

0 

69 

c n rd pass, veobject h 

6188 

3 


0 

0 

0 

0 

2048 

0 

0 

0 

4140 

70 

cnrdequi’pmenth 

7125 

23 


0 

0 

0 

890 

600 

1800 

1244 

2264 

327 

71 

pvpmodule.h 

7597 

7 

1 

0 

1068 

0 

1077 

1086 

3267 

1099 

0 

0 

72 

kontroteventh 

7919 

4 


0 

0 

0 

1966 

1966 

1983 

0 

0 

2004 

73 

rdrtem.h 

8459 

9 


0 

0 

0 

2289 

769 

2154 

1077 

1080 

1090 

74 

cnuser.h 

10660 

9 


0 

1166 

3516 

1183 

1192 

0 

0 

3603 

0 

75 

graphicsystem.h 

10708 

4 


0 

0 

0 

0 

0 

0 

2676 

8032 

0 

76 J 

cnrdactonschpth 

11156 

5 


0 

2207 

2213 

2228 

0 

2253 

2255 

0 

0 

77 

securityc^e^tmi.h 

11287 

5 


0 

2235 

2235 

2250 

0 

2272 

0 

0 

2295 

78 

rdcofsonobjecth 

13170 

€ 


0 

0 

0 

4346 

4381 

0 

0 

0 

4443 

79 

^controlh 

15626 

8 


3842 

0 

1947 

3894 

1964 

1979 

0 

0 

2000 

80 ' 

cnrdobjecth 

15647 

7 


0 

0 

2209 

2216 

4467 

0 

2247 

2251 

2257 

81 

rdpopupvki ndov^’.h 

17690 

19 


0 

902 

0 

3677 

1855 

3726 

3750 

941 

2839 

82 

cnrdanimat'on.h 

20185 

9 


0 

4446 

0 

4472 

9003 

2264 

0 

0 

0 

83 

rdcommon.h 

28924 

11 

1 

0 

2599 

5219 

10496 

0 

5296 

2652 

0 

2660 

84 

rdcharacter.h 

29870 

30 


972 

974 

390-6 

2965 

1987 

7956 

1992 

4040 

5078 

85 

rdactveob-ecth 

31002 

18 


0 

1691 

5089 

1705 

6887 

1731 

3466 

5205 

5228 

86 

rdmonster.h 

33218 

21 


0 

1552 

7794 

3140 

1579 

6369 

6374 

3192 

3218 


Analyzing change impact 
along the time 


Average impact of 
irdmonster.h: 3178 

• Change of irdmonster . h caused 
3178 cpp files to be required to 
recompile for a week in average 

• Huge side effect along the time 

Began to work on major 
bottlenecks 

• Popup window 

• CNRDAnimation 


IIEK 
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Focusing on hotspot was quite efficient 


• Large gain with low cost 

• When focusing on hotspot by changes and change impacts over time 

• Build tinne cut in half with very small effort 


Effective even without understanding the whole system or source code 


GDC'" 

Fighting against Legacy - 
Running game for a Long Time 

Establishing Integration Process while in live 
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GDC^ 


Another important problem: 

Branches were not being integrated 


• Was a small company when launching 

• Dev team and the industry didn't have much experience of 
launching service overseas 1 0 years ago 

• Made separate branches for service regions and separate teams 
worked on them 


lllk 
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Overseas live dev team suffered from merge hell 



Work on authenticaiton, billing, etc for China Launch 

Launch! (First Release) 

China Live Team develops China content and events 
Fork main trunk source to get updated for major contents 

Merge back previous China content 

Heavy release with risk & trouble-shooting 

China Live Team develops China content and events 

Fork main trunk source to get updated for major contents 

Merge back previous China content 

Heavy release with risk & trouble-shooting 

China Live Team develops China content and events 
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Gone too far, too late 


• Cost and tinne to merge along the main branch for a major update 
constantly increased to 4 months to work 

• “I feel like launching a new game every 6 months" 

• Missed golden time to integrate 

• Decided to establish an integrate process 

• unsustainable process, highly likely not to make the major update within 2~3 years 


iirk 
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70 Programmers working on a single project 









GDC 
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rAM 


1 Development Teams on a Project 


Domestic 

Development 


Server Dev 


Client Dev 


Global 

(International) 

Developement 


China Live Dev 


Japan Live Dev 


Tools & Infra 


Security 

Tools/Library 


Contents Dev 


V 


V 










Growing 

superfast 

• >100,000 revisions 

• New source comitted 
every 1~2 min 

• Beginningof 2011: 5,000 files 

• End of 2012: 10,000 files 

• End of 2013: 11,000 files 
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Tuesday)»)<«J»!((WB^5»|iS 10:31:39 


Growing superfast: 

Source commit 
visualization 
in a single day 

*gource visualization 
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19 official releases every month 


• Korea: 4 releases / month 

• Japan: 2 releases / month 

• China: 2~4 releases / month 

• U.S: 2 releases / month 
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Replacing wheels of a car while driving 

Establishing integration process while live operation 
Must not stop update releases with marketing plans 


GDC 
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Long Story, Short Takeaways 

• Took 2~3 years to establish integrating pipeline 

• as a every 7 team's merge/test/integrating process 

• DRY - Duplication is Evil 

• Do not underestimate the force of darkness and the evil 

• To make large size of change / refactoring 

• Can't do big thing by self or with small number of people 

• Establish Roadmap in big picture, Share Vision 

• Make people talk and communicate about the long term need, and goal 

• Cooperate with influential people and leaders to follow the vision and goal 

• Persuade the opponent side 

• Plan to Minimize Risk as much as possible 

• Establish Roles and Responsibilities 

• especially for parts that have overlapped responsibilities between different organizations 

• Earn small-wins. Leverage them 



iiik 
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Short Tips : 3 ways to merge in SVN 
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Short tips: with integration branch, 

you can choose timing and speed to merge, integrate 


trunk integration global (work) japan (live) china (live) 
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Established Code Integration Process 




GDC*" 

Mitigating Technical Risk 
of Many Games for a long time 

(in reusable, efficient way) 


What Nexon is doing more 

For Maintaining Success for the Long Term 
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Nexon's Aged Live Games: 

Games that Nexon is Running for a Long Time 


Title 

Release date 

Age (as of Mar 2016) 


Maple Story 

Apr 2003 

13 yrs 

\ 

Mabinogi 

Jun 2004 

12yrs 


Atlantica 

Feb 2008 

7 yrs 


Shaiya 

Jun 2006 

10 yrs 


Vindictus 

Jan 2010 

5 yrs 


Counter Strike Online 

Dec 2007 

9 yrs 


Kart Rider 

Jun 2004 

12 yrs 


Crazy Arcade BnB 

Oct 2001 


>1 1— 


20 years! 


Nexus: The Kingdom of the Winds 

Apr 1 996 


Dungeon and Fighter 

Aug 2005 


# 

_L 


1 1 yij 

— - 

/ 


>60 services 

in different countries 






Most games suffered from similar problems 


• Most live teams were individually solving similar problems 

• Some of the solutions were not working properly 

• In 2014, Made ‘Live Infra Technology Team' 

• Support 1 4 titles (60+ live game services) 

• Create efficient solutions to solve common technical problems 

• Focus to help live game services, while persuing Reusable/Scalability 

• Help future titles/services launches 


Important problems in running live service 


• For Game as a Service, mitigating risk is crucial to maintain its success 

• Kind of common risks in game services: 

• Technical Risks (=Stability Issues) 

• Crash often 

• Memory error 

• Too laggy to play 

• Preventing client hacking (especially in PvPs) 

• Server problems (cannot log in, unstable server) 

• Other risks 

• Loss of Virtual Properties (items, achievements, etc) 

• Negative press 
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Stability Problems are important 

• Stability Problems are big in online games because: 

• Users will leave the game when stability problem last for days 

• Users usually don't get a choice to upgrade or rollback 

• Stability unresolved = Dead Game 

^ Ensuring minimum stability baseline is very important 

• Major Stability Issues 

• Crashes 

• Memory Problems 

• Lag Issues 

• Unstable Server 

• Hacking 


iiik 
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How should we deal with the Risks 


Recognize | 

Define/Prioritize 

1 Find Cause 

W Act (Fix) " 

1 

• Define Problem 

1 

• Develop/Build/Test 

L M 

• Determine Severity 

^ t 

• Deliver/Deploy 



The lesser time & cost, The better 
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Major Stability Issues 


Recognize 


• Crashes 

• Memory Problems 

• Lag I ssues 

• U j-gii^rg scope) 



I 
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Many of old PC games 


used manual tools 


Recognize B Prioritize I Find Cause I Fix 


• Collecting and Analyzing data was costful 

• Had to collect dump across servers 

• Required a lot of manual work and time to collect and analyze 

• Publisher, QA team requested crash reports if issued but not for every releases 


A 

B C 

D 

i E 

0181FAA4 

443240 const NiTArray< class NiDX9AdapterDesc ‘class NiTMalloclnterface<class NiDX9AdapterDesc 

const NiTArray< class NSBD3D10TextureStage *, class NiTMalloclnterface<class NSBD3D10Textu 

const NiTPrimitiveArray< class NiTextureEffect const *> 

01821748 

116488 const NiSPStream::'vftable' 



01819280 

114522 const NiBSplineCompTrans^mEvaluatonfvftable’ 



0181F568 

106653 const NiDX9LockableDataStream<class StaticLockPolicy<class NiDX9DataStream> >::'vftable’{for NiObject'} 


0181F540 

106652 const NiDX9LockableDataStream<class StaticLockPolicy< class NiDX9DataStream> >::'vftable'{for NiSPDataStream'} 


01817750 

95540 const NiTransformEvaluaton:'vftable' 



01837B30 

92314 const NiTSPFixedlnput<class NiMatrix3x4>::'vftable' 

const NiTSPStream< class NiTransform * *>::'vftable' 

const NiTSPFixedlnput< struct NiPSSimulatorFinalKernt 

01761A04 

77452 const cltPltemInfo_Client_SolNPC::'vftable' 



01821DCC 

60451 const NiTPointerList< class NiPointer<class NiRoom> >::'vftable' 

const NiTPointerList< class NiPointer<class NiDX92DBufferData> >::'vftable' 

const NiTPointerList< class NiPointer<class NSBShader 

01818298 

55509 const NiTransformData::'vftable' 



01821CF0 

55078 const NiTListBase< class NiTPointerAllocator< unsigned int>, class NiDynamicEffect *>::'vftable' 

const NiTListBase<class NiTPointerAllocator< unsigned int>, class NSBD3D10StateGroup::NSBDi 

const NiTListBase< class NiTPointerAllocator< unsigned 

018264E4 

54653 const NiSemanticAdapterTable::'vftable' 



018264E0 

54647 const NiTArray< class NiSemanticAdapterTable::SemanticMapping, class NiTNewlnterface< class konst NiTObjectArray< class NiSemanticAdapterTable::SemanticMapping>::'vftable’ 


01834810 

47497 const NiMesh::'vftable' 



018335E4 

37680 const NiTObjectArray< class NiPointer<class NiShaderDeso >::'vftable' 

const NiTObjectArray< class NiPointer< class NiControllerSequence> >:;'vftable' 

const NiTArray<class NiPointer<class NSBUserDefined 

01823098 

21206 const NiMaterialPropertyfvftable' 



01822A7C 

17024 const NiTexturingProperty::Map::'vftable' 



01821724 

16259 const NiSPTask::'vftable' 



018216E0 

16256 const NiSPTasklmpl::'vftable' 



01822AB0 

14496 const NiTexturingProperty::'vftable' 



0175B334 

12933 const NiRefObject::'vftable' 



01833F20 

12852 const NiStringExtraData::'vftable' 



01822088 

11831 const NiSourceTexture-'vftable' 



01860DF4 

11600 const NiTPointerList< class NilnputDevice::ControlDesc *>::'vftable' 

const NiTPointerList< class NiDynamicEffect *>::'vftable' 

const NiTPointerList< struct IUnknown *>::'vftable' | 

01818068 

11098 const NiEvaluatorSPData::'vftable' 









GDC 




GAME DEVELOPERS CONFERENCE March 14-18, 2016 ’ Expo: March 16-18, 2016 #GDC18 




Crash Reporting as a Service 


Recognize B Prioritize I Find Cause ■ Fix 


• For game as a services, Crash Reporting also should be a service 


• everybody should be able to crash status 
not only developers, but also QA team 

• How to make it more efficient and effective 

to live game services? 


*ln mobile games, App Stores usually have developers' console for crashes 







Prioritize 


Determining Severity Properly 


Recognize 


Find Cause I Fix 


• There're many noises to determine severity properly 

• Out of memory crashes, C Runtime Errors weren't caught in general 

• Many crashes were being occurred after pressing Exit Game button - 
priorities different 

• Crash count fluctuated with DAU — 

13th. 

Canvas+4)x799 

MapleStorylCBitmapStrNuirber: :^BitmapStrNumber+ei<‘j 
6 C£S> ntdll!RtlplnsertUCRBlocl<-t-i)x4 

12th. 

HapleStorylCUlToolrip: :~CUIToolTip+0xba8 [uitooltip.cpp # B6B] 

6C C6S. Haplei>tory!CCtrlButton: :-CCtrlButton+0xl0d [ctrlbutton.cpp g b8] 

HapleStorylCCtrlButton: scalar deleting destructor '+0x 


16th. 

MapleStory!ZRecyclableAvBuffer<ZRe-FCountedDummy<_cow_ptr_t<_co(ii_lllD<lWzCanvas,&_GUID_/600dc6c_9_ 
5 xss. «apleStorylZList<_cofli_ptr_t<_coir_IlIO<lWzCanvas,&_GUlO_7680dc6c_93i8_4bff_9624_Sb0fSc01179e> > >_ 
MapleStory!ZList<_coffl_ptr_t<_coir_lllO<lWzCanvas,&_GUlD_7600dc6c_9328_4bf'f_9624_bb0-fbc011/9e> > >_ 


17th. 
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Monitoring Crashes 

Is it still unstable? 


• After 1 2 Nov update, 

# of crashes increased a lot 

• Released 13 Nov fix patch 

• Still high on 14 Nov 

• Is it still unstable? 
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Monitoring Crashes 

Is it still unstable? 


• Active Users increased a lot 
after 11/13 patch 


Recognize B Prioritize I Find Cause ■ Fix 
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Monitoring Crashes 

Is it still unstable? 


• Tried to reduce fluctuation noise 
regardless of active user volume 

• Crashes per session - better, still noisy 

• Crashes per hour - represents very well 

• Both session and time are related 

but per unit time indicated stability better 
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Avoiding Boiling Frog Syndrome 


Recognize B Prioritize I Find Cause ■ Fix 




vei 

vcr 2016-01-06 647 Slieni 

2016-01 

-06 64f' 






ver 2016-01 

-11 649 






















_ A 

" s 


- 

1 

i 

L 

f , . 




A 

1 

- 1 

1 . 



Jan. 04 Jan. 05 Jan. 06 Jan. 07 

Jan. 08 Jan. 09 Jan 

.11 Jan 

.12 Jan. 

1 3 Jan 

. 14 Jan. 

1 5 Jan 

16 








Avoiding Boiling Frog Syndrome 


Recognize B Prioritize I Find Cause ■ Fix 
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Made a Crash Report as a Service 


CrashReporter v1.0 


O Manage * \s MapleStory Korea ' 



^ MapleStory Korea 




Lilli Dump Trends l*l Crash Count liMACU 


Q < 2016-3-12 > 



This Week 
(per hour) 


— Last 6 Hour — 

(per 10 minutes) 








1 


n 

10 3- 



1 

£ 5 

^ ^ ■■ 
-.y 

V i' ■ 

s 1 

- Q 



Mar.06 Mar.07 Mar.08 Mi 

2 weeks ago Last week ■ Analysis Failure 

ir.09 Mar. 10 

Crashes per time unit 

Mar.11 Mar.12 9PM 12. Mar 

Last week This week 



Recognize I Prioritize 


VerMon : M^Stofy_253 (201&4)2-25 : 11531 )« 


j= Most Reported Callstacks 


Search: 


Rank 

Call Stack 


1st. 

-■-£fnat_trrr" ■ 


NEW 

Canvas-F0x799 

MapleStory ! CWvsContext : : ~CWvsContext+0x436 [wvscontext . cpp @ 863 ] 


41S0 

MapleStory ! CWvsContext scalar deleting destructor '+ex 


(36.3%) 


show detail... 
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For many game services 


Recognize B Prioritize I Find Cause ■ Fix 


• 14 projects 

• 54 crash dashboards 

• Not only client 

• But also 

• Game Servers 

• Dedicated Servers 

• Middlewares 
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Made a Crash Report as a Service 

CrashReporter V1 .0 O Manage » is MapleStory Korea » 


^ MapleStory Korea 


Lilli Dump Trends 

J 1*1 Crash Count [*IACU 




Q < 2016-3-12 > 




This Week 
(per hour) 


= 

Last 6 Hour — 

(per 10 minutes) 







n 

10 1 
s 1 

e 

1 

^ 5 

u ! ^ 

-.y 

. 



V i'. 



Mar.06 Mar.07 Mar.08 Mar.09 Mar. 10 Ma 

2 weeks ago Last week ■ Analysis Failure Crashes per time unit 

r.11 Uar.12 

9PM 12. Mar 

Last week This week 


VerMon : M^Stofy_253 (201&4)2-25 : 11531 )« 


j= Most Reported Callstacks 


Search: 

Rank Call Stack 

-■-£fnat_trrr " ■ 

Canvas+0x799 

MapleStory ! CWvsContext : : ~CWvsContext+0x436 [wvscontext . cpp @ 863 ] 

MapleStory ! CWvsContext scalar deleting destructor '+ex 

show detail... 


1st. 

NEW 

41S0 

(36.3%) 
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Details 


^ MapleStory Korea 



~ Most Reported Callstacks 


Search: 

Rank 

Call Stack 


1st. 

iiajow bjcfer>iimcou(ex.at»i.‘:K j 



KtRNELBASb!RaiseException4«xS 


iUSlAsi 

MapleStory ! _Cxx T hi'owExcept ion-sDxdS 

[throw. cpp § 161] 


MapleStory ! _coe_raise_er ror4Dx 50 

[conraise.cpp #18] 

2nd. 

uctrirai.iicass,vuxxii(>i(eiLM«ew-. 



Gr2D_DX9+0xe93 


sn.sn.., 

PCOM+0Xl62b 



PCOM+0Xl62b 


2rd. 

...B.iioi..cass.vioiAi: . . 



Canvas-f0x799 


3S53S0.I 

ntdll ! RtlplnsertUCRBlock-s0x4 



ntdll ! RtlplnsertUCRBlock-h0x4 


2th. 

txct»TiaM_«cass_vjaL»ti--r<e neeaieii) 



Cr2D_DX9+0xe91 


132.32^1 

PCOhts0xl62b 



PC0M^xl62b 


5th. 

ucteiio. >caii m „mf-] 



Gr2D_OX9+0xe91 


2*iaTHi 

PCO»s0xl62b 



PCOMr0xl62b 


6th. 

— t , 



MapleStory ICWndMan : ; s_Updater0x4a 

[wndman.cpp # 702] 

220 3 1S) 

MapleStory ! ^dyn_tls_init_callback 

<PERF> (MapleStory+0xle38706 


MapleStory! ^dyn_tls_init_callback 

<PERF> (MapleStory+0xle410e9 

7th. 

-I ; 



igduBd32+0x4f5 



igduadj2+0xSlc 


8th. 

.... • .... -.r. ,-w^. 



MapleStory !ZRecyclableAvBuffer<ZMap<long, ZRefcCltemlnfo: :EQUlPlT£M>,long>: :_PAlR,16,ZMap<long,ZR_ 

12K12SI 

MapleStory ! ZMap<long, ZRef <Cltemlnfc 

i: :EQ01PlTEM>,long>: :Insert+0x6a [_zcoll.ini # 1056] 


MapleStory ICltemlnfo: ;RegisterEquiplteiilnfo-i0xici [iteminfo.cpp @ 21S0] 

9th. 

t»CH-uc«ca», - 



KERNELBASE+0xb3fc 


105(1-0^1 

MapleStory ! _Cxx T hrowExcept ionAdx48 

[throw. cpp # 161] 


MapleStory !_coa_raise_error40x 30 

[comraise.cpp # 18] 

10th. 

US VUO) ncauyuxxi iax*>L*MMes ) 


• M 

0xl48eS 


tsots. 

0x10044 



Recognize 


Prioritize 


Fix 


A Details 
it Rank 

8th. ( 1.16% ) 

it Exception Code 

EXCEPTION_ACCESS_VIOLATION|OxcO<KIOOOS) 

U Fnt Occurrence 
vemon : MsftecxrfjiS 
Date: 2015-11-12 09rtt is 


Last weak This VVaekiDump ' AlZI enh.) 



<? V® v' s’* 

^ 


it Cat Stack 



Search: | | 

Crash 

Download 

Account 

Character 

IP 

Usage^ 

Call 

suck 

201WIS-13 

00:30:22 


tlsdlekgm)72 


58.127.168.183 

1225 MB 

■ 

20194)3-13 

00:16:19 

(U®® 

c"tmr@'" 

S=f 

118 39.115.91 

1279 MB 

■ 

201903-13 

00:12:23 

;^® ® 

h"55‘e"‘ 

j 


118.130.169.177 

1002 MB 

■ 

201903-13 

00:06:01 

®|'a'| ® 


RkH 

121.176.174.246 

1015 MB 

■ 

201903-13 

00:03:32 

® 1® ' K 

bantel 

j 

80 -i- 

j 

211.202.168 23 

633 MB 

■ 

201903-12 

22:59:03 

® a ! s; 

I•■jm961129•e•" 

sfaj3-oin 

122.32 185 104 

. 

1263 MB 

■ 

201903-12 

®® K 


xxekzmfkdixx 

219.248.94 248 

1114 MB 

■ 









Details 


^ MapleStory Korea 


Recognize 


ti Rank 

Sth. ( 1 . 16 % ) 

bE Esceplbn Coda 

EXC EFT IOH_ACC E S S_VIOL AT lOMjflKcCOflflflflS] 

bi Firsi GocuTfence 
VeffBfciii : : M^ 6 SlQr>_ 2 iS 
Date: 2015- 11-12 0?:^::15 


bi Call Slack 
Mapleitory 1 ZRecyc La blGfivBufFe r<ZMap < lor 
napleitorylZHap^longjZRef <CItGiriinFa : : Ei; 
HapLeitorylOtemlnfo: : RagisterEquipltei 
HapleitorylCltemlnfo: : □GtEquipItem-05<L] 
WapLeitory 1 Clteml nfo : : Get Eq ui pped E lotit 
WapLeitorylCflvatar;: : NotifyflvatarPtodifit 
WapLeitorylCflvatar;: :lnit-H0xaf [avatar 
HapLeitoryltflvatarMagaphone : ::OnCraate-i-E 
WapLeitoryltWnd : ::CrGateWnd-H3sbBa[wn(I.CF 
WapLeitorylCflvatarMagaphone : iCflvatarJtei 
WapleitorylCiHvstontaxt: : DnSetAvatarMegE 
WapleitorylCWvstontaxt: : QnPaclcet-Hax94e 
WapLeitory 1 dyn_tl s_in it_cal Lbac k ^ P L F 


Lutwvok This Waak-Dump.'AIS- t4h.l 


Dump Cdui* 


4 



[} 

^ ^ 

^ 








Dump List 


>irlnfc: :(Q||I«TtH>,l«lfa: tin 

-tn Y**— T-r-itr1r I 


Crash 

Time 

Download 

Account 

Character 

IP 

Meomry 
J sage 

Call 

atack 

2010 ^ 3^13 

00 : 3 fl :22 

[IJEll 

<bdlekgm |72 


53 . 127 . 103.133 

12251 MB 

■ 

2010 - 03^13 

00 : 10:19 

[U[1]E 



113 . 39 . 115.91 

1279 MB 

■ 

2 O 10 -O 3 -T 3 

00 : 12:23 


h“ 55 '@'" 


113 . 130 . 109.177 

1002 MB 

■ 

2010 - 03-13 

00 : 00:01 

[UEB 

- 

Ri!“ 

121 . 170 . 174.246 

1015 MB 

■ 

2010 - 03-13 

00 : 03:32 

□ 1 , 

bainell 

ac-E-- 

211 . 202 . 103.23 

633 MB 

■ 

2010 - 03-12 

22 : 59:03 

EEd] 

|■■Jm 901120 ‘@■■■ 


122 . 32 . 135.104 

1263 M B 

■ 

2010 - 03-12 
22 : 2^:20 

□ 1 

- 

xxekzmlkdbx 

219 . 243 . 94.243 

1114 MB 

■ 



rfc>iiMr«ti|itiiiiu>niT rti | wjt»B;|iphafi».cpi> f « 

AvatarMecapham-axUf |MVicontiitZ..cpp # U8K]' 

Iwv:o3ntCKt.cpp f HJ9] 


[_icsii.ini ? iifS4] 

:pp » H»] 


m-e- 



Kd'DiB UZ.IZIB.IC 
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Monitoring Crashes 

Adding Notification and Aiarms 


Recognize I Prioritize 


Automatically detects patch 
and sends notificaion mail 
after 2 hours 

Some kind of dumps 
are sent immediately 
( Server dumps, from internal test ) 




2016-01-01 4:18 

[infratech] 

[3EHA|e|£E|] iiH^;| ^ 


0 .cso^ # 2 t 0 |oie 5 i£ [cso-cSent]: 0 .2!“2t2|«g [infratech] 
* s wiijs Ai^a« m Ahsss otiijgjua, 


[p hbhaihisei 
iB||0|X|S Ol-t 


Hsaissi (VER.20l5_l2_17.03/Client)mao| nHSISIK^qcr. 
ms°l SHS ^'■4 Sa|2K201G-01-01 02:i7;oi)hs 2a|zj-o| AIUS5 4AHI assSHaua. 




Si^il 

(1AI2f9»l) 



12IC26V I2H27II 12«28V I2«29tl 1200)1 12tf3l)! OlSOlV 


2? 3 i?a aaiAi4^ 


.01/Client (2015-12 


i= analog 
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Monitoring Crashes 

Adding Notification and Aiarms 


Recognize B Prioritize I Find Cause 


If crash rate is increased 
a lot more than usual, 
it immediately sends mail 
within 2 hours 



2015-12-16 (4=) 2* 10:07 

[infratech] 
[ 3 ENA 1 EISEI] nHA| aEHAiif 


Alg m nnfratech] 

05 £l£ 7 ^ 0 |x|x|g ayisMQ. 


♦ s misjs Alia* s«N AhSHs iHisj£;ua. 


tp aa||A|&|sEi 
-Jk 2 flO|A|£ 0 |^ 


H5a| = 2| (VER_EU_I512.07.57523-I51215-I81437/Client)m30| IiHA|E|5!6UU. 
ms ? 1 AIL7 40 SOI AiyaiH Ef4<90 41)91 3eflA|7J SUSlfiUCI. (INAI ^ 2A|L^S°^ «°J S3 BH7H4:89 7H) 


* 3 . 3 \x\ y\^ nfl^l 3 |« 21 % 7 H^^go||A|^ i||x|g ul 2 e| aeflA| § 7 h®-^o|| qjsH 

se 3c!HA| S20| StHUfe §s| g"CH| °|SHAHE 35HAI 7n^7[ arois 4^ SIHHS 4f3 mSfMCf. 


I — I ^-A^ I — J I I t3 

iM SH aan (i9» 1 iaizi> 


si^ti = se 6 Ai 7 i = 

(IAIE198I) — (I0»9«) — 






i’\ 

I \ 

lOV 12« 




!ll« 12s 

I2« 12«I3« 12KUV 12V 

ISi; 121; 161/ 


2? a 1?S ■»6!S71 3«AI4 




m 


aH2i±^ 


Fix 




$?l 1±9! 

1 ^ cplushook-fOxSdl 
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Monitoring Crashes: 

Sometimes Call Stack is not enough 


Recognize I Prioritize 


• • 


Find Cause I Fix 


• When broken state caused crash later 

• Heap Corruption 

• Dangling Pointer 

• If it is not reproduced, it is sometimes very hard to find cause 
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Monitoring Crashes: 

Provided more detailed informations 


Recognize B Prioritize I Find Cause ■ Fix 





^ Decode UTF-16 ✓ Decode UTF-8 ^ Decode CP949 / Decode Ascii 


Bbaseinfo.json 

BaseCharacterType : "Arisha" 

CHAR_NAME : "IKrewella" 

CID : 63542997%36306600 
CPUJNFO : "InteKR) Celeron(R) CPU G1850 @ ; 
CRASH_BEHAVIOR_BITFLAG : 13 
CR_VERSION ; "VI. 2.0.1" 
DUMP_TYPE_BITFLAG ; 3 
ENCODING : "UNICODE" 

Level ; 80 

MEMORY_PEAK_USAGE : "1403 MB" 
MEMORY_PHYSICAL : "3714 MB" 
MEMORY_USAGE : "727 MB" 
OS_BUILDNUMBER : "9200" 
OS_HAS_CHINAIME : false 
OS_LOCALE_SYSTEM : "t_ROi(CHtry ^)" 
OS_LOCALE_USER "tr^Oi(CHtrei^)" 
OS_PRODUCTTYPE : "1" 
OS_SERVICEPACK_MAJOR ; "0" 
OS_SERVICEPACK_MINOR ; "0" 
OS_SUITEMASK : "0x00000300" 
OS_VERSION_MAJOR : "6" 
OS_VERSION_MINOR ; "2" 
OS_VERSION_NAME(log_viewerSg) ; "Windowi 
OS_WOW64 : false 
PLAY_TIME : 26079 
UTC CRASH : "2016-03-11 15:01:04" 


*Type of data to collect varies for different game and countries under game policy and user agreements. 
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Screenshots 



give clues 




Recognize B Prioritize I Find Cause ■ Fix 




*Type of data to collect varies for different game and countries under game policy and user agreements. 


[88 6 












GAME DEVELOPERS CONFERENCE March 14-18, 2016 ’ Expo: March 16-18, 2016 #GDC18 


GDC 




Monitoring Crashes: 

Don't require to ask player to send client log 


Recognize B Prioritize I Find Cause 


L 03/12/2016 
L 03/12/2016 

00:41:10 

00:41:10 

Log file started (version "6331") 

L 03/12/2016 

00:41:10 


L 03/12/2016 

00:41:11 

Shader_Connect 

L 03/12/2016 

00:41:11 

ConnectStudioRender 

L 03/12/2016 ■ 

- 00:41:11 

IDataCacheFactory 

L 03/12/2016 • 

- 00:41:11 

IPhysicsFactory 

L 03/12/2016 ■ 

- 00:41:11 

IMDLCacheFactory 

L 03/12/2016 ■ 

- 00:41:11 

IMatSystemSurfaceFactory 

L 03/12/2016 

- 00:41:11 

lAviFactory 

L 03/12/2016 ■ 

- 00:41:11 

IHammer Factory 

L 03/12/2016 ■ 

- 00:41:11 

Connectf^LCacheNotify 

L 03/12/2016 • 

- 00:41:12 

Enginelnit 

L 03/12/2016 ■ 

- 00:41:12 

RunListenServer 

L 03/12/2016 ■ 

- 00:41:12 

Tracelnit : COM_InitFilesystem( pModName ) 

L 03/12/2016 ■ 

- 00:41:13 

Tracelnit : MapReslistQenerator_Init() 

L 03/12/2016 ■ 

- 00:41:13 

Tracelnit : DevShotGenerator_Init() 

L 03/12/2016 ■ 

- 00:41:13 

materials- >ModInit 

L 03/12/2016 ■ 

- 00:41:13 

Host_ReadPreStartupConfiguration 

L 03/12/2016 • 

- 00:41:14 

NT^COGame : : _nmman . Init( ) 

L 03/12/2016 ■ 

- 00:41:19 

NMCOGame : : _nimnan . Set LocaleAndRegion ( ) 

L 03/12/2016 ■ 

- 00:41:20 

NMCOGame: :_ntnman. Initialize () 

L 03/12/2016 ■ 

- 00:41:24 

NMCOGame : : NMGameLogManager . Initialize () 

L 03/12/2016 ■ 

- 00:41:24 

nmcoGame - > I nit () 

L 03/12/2016 ■ 

- 00:41:24 

endPointNetwork->Init() 

L 03/12/2016 ■ 

- 00:41:24 

mmoEndPointNetwork->Init() 

L 03/12/2016 • 

- 00:41:24 

AppSystem : : RegisterHandler<ServerCmdMessage> 

L 03/12/2016 ■ 

- 00:41:25 

AppSystem : : RegisterHandler<ClientCmdHandler> 

L 03/12/2016 ■ 

- 00:41:25 

AppSystem: : RegisterHandler<SyncFeatureNatrixMessageHandler> 

L 03/12/2016 ■ 

- 00:41:25 

AppSystem : : RegisterHandler cNGSecurityMessageHandler > 



I 



*Type of data to collect varies for different game and countries under game policy and user agreements 
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Jupyter connected 


• Can do any other analysis just-in-time: 

(powered by python & Jupyter) 

• How much of RAM do crashed users have 

• Check whether crashes is occurring more to specific users 



In [6]: from collections import Counter 

counter = Counter([dump[ ’char_name’ ] for dump in dumps]) 
labels, values = zip(*[(k, v) for k,v in counter. most_common( 20)]) 
indexes = np.arange(len(labels)) 

pit. bar (indexes, values) 
plt.xticks(indexes, labels) 
plt.showO 

20 


15 


lllllllllllllllllll 


in [ ]: 
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Monitoring Crashes 

Helped Live Service & Live dev team! 


Recognize B Prioritize I Find Cause I Fix 


Helped live dev team to concentrate more 

on developing game 
without being distracted with live issues 
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Major Stability Issues 


• Crashes 

• Memory Problems 

• Lag Issues 


GDC 
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Memory Problem: 

Inevitable when running game for a long time 

• As contents accumulated, most games suffered out of memory problem 
even with plenty of RAMs 

• 32 bit process memory address space limitation: 2GB 

• 2GB is not enough for many module images and resources including Ul images, textures, animations and sounds 

• /LARGEADDRESSAWARE helped, but an one-time opportunity 

(allows process to use 3GB of user memory address) 

• Especially with many of high-res textures, or many displayed characters: 

• 2D animation with many frames 

• MMORPG with various character costumes 

• Lots of background/UI/lllustration images 



lllk 
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Memory Problem: 

Many of Crashes are because of Out of Memory 

• Most game projects operating more than 5 years had plenty of out-of-memory 
problems 


KERNELBASE ! RaiseException+0x5 

engine !Sys_Error+0xc0 [sys_dll.cpp @ 401] 

engine !MemAllocFailHandler+0^ [sys_dll2.cpp § 2038] 


.. ;_EXCEPUONCODMi 


kernel32 ! RaiseException+0x5 

engine !Sys_Error+0xc0 [sysdll.cpp @ 401] 

engine ! MemAllocFailHandler-h0x32 [sys_dll2.cpp @ 2038] 

KERNELBASE ! RaiseException+0x5 

engine !Sys_Error+0xc0 [sys_dll.cpp @ 401] 

engine IMemAnocFa^Hand^^ [sys_dll2.cpp § 2038] 


Ui Dump Trends j.i ' 


KERNELBASE iRair-*- — 

Client ! esi : : et, Client ! esl : : etc : : +0x19 [ es! 

Client!esl::SEI . +0X94 [es! 

Client Ihandlej 

client ! handlej Client ! handle_program_memory_deplet ion+0xld 
msvcpieoistd: : Client ! handle_program_memory_depletion+0xe 


3 


+0x19 [esletc.cpp @ 1926] 

:94 [eslseh.cpp § 90] 

>tion+0xld [mabinogiinain.cpp ^ 136] 

ition+0xe [mabinogiinain.cpp ^ 136] 


msvcriee ! opera- 
client Ipleione 


;= Most Reported Callstacks 


KERNELBASE !RdiseException4-0,x5 

C lient ! esl : :etc : : rai*«_oxc«ption+0xl9 [esletc.cpp g 1926] I 

Clientlesl: :SEH: :Raii*Exc€ption+0x94 [eslseh.cpp f 90] I 


KERNELBASE !RaiseException*Cx5 

Client'esl: :etc: : raii«_«xe«ptiontOxl9 [esletc.cpp £ 1926] 

Client !esi : :SEH: :Rjis*Exc«ption+C>:94 [eslseh.cpp £ 90] 


msvcpl00!std: : 

>e ! opera- 

pieione nisvcrl00 !_callnewh+0xl 
pieione msvcrl00! operator new+0xl 
pieione client ! pleione : :mesh vb:: 

pieione ” 

pieione Client ! pleione : : sspt_pass : : 


pieione client ! pleione : :mesh_vb: : +0x85 [plr_primitivemesh2base.cpp @ 219] 

pleione 

pleione Client ! pleione :: sspt_pass : : +0x28e [plr_staticpartitiontree_r€ 

pleione: iCStaticPartitionTree: : -fexSe [plr_staticpartitiontreecollapse.cpp ^ 51] 

pleione: :scene_context: : +0x261 [plr_scenepretransform.cpp § 1142] 

pleione: :CScene: : +0xc37 [plr_scenepretransfonn.cpp ^ 946] 

pleione: :CScene: : +0xle9 [plr_scenepretransforn.cpp ^ 194] 

pleione :: eWorld : : +0x56d [plw_worldrender.cpp g 373] 

pleione: :CGlobalConsole: : +0x58a [pleioneglobalconsolerender.cpp § 1543] 

pleione: :CAccount: : +0xed [pleioneaccount.cpp § 1236] 

pleione: :CPleione: : +0x125 [pleione. epp @ 410] 

pleione: :CPleione: : +0x122 [pleione. epp § 114] 

Main+0x65a [mabinogimain.cpp § 1022] 

MainWithFileSystem+0xle2 [mabinoginain.cpp @ 1176] 
wWinMain+0x605 [mabinogiinain.cpp § 1531] 

_tmainCRTStartup+0xl58 [crtexe.c § 547] 
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Memory Problem: “ 

Tracking Memory wasn't easy for PCs 

• Especially for many old games didn't have good engine with 
resource/memory tracking functionality. 



• Reports based on Task Manager used to have noises 

• XP Task Manager displayed "working set" which doesn't count paged-out memories 

• Size gap was big between "Private Bytes" and tracked memory size, 

Needed to narrow the gap 

• Using ‘■_CrtSetAllocHook^ and Overriding new didn't have full coverage (e.g. DirectX, GUI middlewares, Sound library) 

• Performance Counter did not represent application level memory usage, but showed memory page usage 

• did not count: free & allocate again, pooling 

• Couldn't distinguish between graphics, sound or cache usages 
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Memory Problem: 

Limitations Using existing Memory 


Recognize B Prioritize I Find Cause 


Fix 


• Tried Memory Validator, GlowCode, Visual Leak Detector 

• Need to 'play' game, without running too slowly 

• Client Security Solution denied access to client process, including Visual Studio 

• Need to track real-server-situation with many of active users 

• Many problems were not reproduced with test doll characters 

e.g. realm vs. realm battle, village with many people doing many things 

• Should handle large amount of allocation data to track out-of-memory situation 

• Most used a lot of memory to keep result 

• Used to crash because of out of memory while profiling 

• Want to compare certain two points of time in detail 

• To check leaks between certain two point of time (ex. between after second battle and third battle) 

• Or to find cause of the drastic increase 

• Want to track memory-pooled objects 
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Memory Problem: 

Made own Memory Profiler for DNF 



Game Client (32bit Native Application) 


in-process 

Memory Profiler Module 


Profile Result Viewer (64bit .NET application) 


sends 

memory allocation stats 
periodically via UDP 



- hooks HeapAlloc API 

- callback from Memory Pool Manager 


bool LoadAnimation(..) { 

If (Loadlmage(imageName) == false) { 

HandleErrorQ; 
return false; 

} 


LoadAninnation(..) 

LoadImage(..) 

CreateTextureFronnMennory(..) 

HeapAlloc(..) 


1 Rank | 

y^tack 

Total Bytes Allocated 

Total Count of allocation 

Avg. Allocation Size 

1 

^oadAnimation():12 

XXX, XXX, XXX 

x,xxx 

x,xxx,xxx 

J2 

LoadBackground{):23 

xxx,xxx,xxx 

x,xxx 

x,xxx,xxx 

js 


xx,xxx,xxx 

XXX 

x,xxx,xxx 

1 
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Memory Problem: 

Made own Memory Profiler for DNF 


Recognize B Prioritize I Find Cause 


• Project-independent but in-process memory profiler 

• Just one initialize call hooks Win32 HeapAlloc 

• Pooled object was monitored via one line of simple callback 

• Sends allocation stats to aggreator/analyzer server (64-bit .NET application) via UDP periodically 
(fast / no insufficient memory) 

• Runtime switches to trade off performance and detail (fast) 

• Usable in real environment with client security solution 

• Can track external library memory usage (DirectX textures, Ul middlewares, sound library, ..) 

• Can diff memory blocks and its call stack between marked point 

• Very useful to track untracked memory gap between performance counter and in-game memory 
tracking, and various leaks as well 

• Plan: Making a generic memory profiler for live games as a service ready-to-use at any time to 
developers. 

• Process-independent solution - reusable in many live games in many cases. 
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Major Stability Issues 


• Crashes 

• Memory Problems 

• Memory-Lag overlapped Problem 

• Lag Issues 
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Memory, Lag - Overlapped Problem 
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Memory vs. Lag trade-offs 




Shorter Stage Loading Time 

Longer Stage Loading Time 

Less Memory 

More Memory 

Causes Out of memory even with plenty of RAM 
(insufficient 32bit Address Space) 

Causes Thrasing with little RAM 

More in-game Loading Lag 

(Can't use background loading sometimes) 

Less in-game Loading Lag 


Need Preload Resources, 

Need to find their reverse-dependency easily. 
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Recog n i ze I 


Preload manager for DNF 


• Challenge: Hard to determine likely-used resources 

• Didn't have game engine or resource manager with unified resource references 

• Many of resource reference was not in declarative form 

• Suggested to build preload list based on actual resource usage log 

• method used in Bubble Fighter 

• Al/Skill-related resources were especially hard for DNF 

• Some were not bound to single animation 

• DNF had variety of Al/skills (> 2000 skills) which were not represented in declarative form 
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Building Preload List automatically 

• Logged every player's event in Test Server 
for couole of weeks 


pi 

77.500.000 image file loadings 

1.400.000 skill events 
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Building Preload List automatically 


• Performed Cluster Analysis 
with variance of time, and 
associated with likely-related 
Skill or Map ID. 




























Recognize 


Prioritize 


Find Cause 


Building Preload List automatically 

• A lot of noise, couldn't cover 1 00% but very effective 

• Preloading skill resources make players feel far more lag-less than others 
because it's related to control response time in battle 


• Problems / Limitations 

• Lots of noise in skill timing, because of low- 

• Players with low RAM experienced thrashing 

- Added automatically decided option to choose preloading 


• Plan: Generic preload manager module with dashboard service 

• Another General Approach for costly and hard problems 

• A Burden for live dev teams to develop full system as a service and maintain 

• Helpful to many live projects having memory-lag issues 
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GDC^ 


Major Stability Issues 


• Crashes 

• Memory Problems 

• Lag/Slowdown Issues 


Lag/Slowdown issues: 

Hard to recognize and define problem 


Recognize 


Prioritize 


Find Cause 


Fix 


• Lag/Slowdown problem is hard to define a issue because it occurs differently in 
different environment 

• Development Team and QA Team had a good machine with SSD 

• couldn't realize about most of loading / lag issues 

• got a test machine with 1GB RAM /XP 

• Hard to realize how much it is happening to every/certain kind of players 

• Still many XP users 

• especially in other countries like China, Vietnam, .. 

• Not easy to test all the contents with every kind of machine for weekly releases 
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Recognize I Prioritize I Find Cause 


Limitations of Conventional FPS monitoring 


• Normally monitors average FPS 

• Collects FPS a single user session / game session 

• Monitors FPS for all or specific contents 

• Can monitor overall performance degradation for all or specific contents 


Hard to monitor FPS spikes / Frame Lag 
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Recog n i ze I 


Monitoring Lag/slowdown better 

• FPS monitoring to collect FPS distribution per time slot 

• Along with various information 

• Map, Character Type 

• Recent Skills 

• System environments (Graphic option / Screen resolutions) 

• Can monitor overall performance including laggy experiences 

• Still hard to reproduce, find cause and optimize 








Limitations in Finding Cause 

using existing Performance Profiler 


Recognize 


Prioritize 


Find Cause 


• Hard to use conventional profiler in game 

• Client security solution denies whole access from outer process, including Visual Studio 

• Many games had developer-mode profiler 

• Many Had to define profile entries 

• or had limitations in breaking down 

• Hard to reproduce performance degradation 

• Get an old system to reproduce 

• Have to perform several experiments because of side effects like hard disk cache 
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Recog n i ze I 


JYP - Just Yet-another Profiler, actual, y„ot, 

• FPS monitoring with just-in-time Sampling profiler 

• embedded sampled Call Stack info with FPS monitoring 


• Can find performance hotspot when needed 

• When frame rate drops 

• When a player experiences lag 



• Not to be confused with JYP, 
famous K-POP producer 




J.Y.Park Wonder Girls 


iiik 
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JYP - Challenges 

• FPO (Frame pointer omission) 

• Can not perform stack-walking properly with PDB 

• Requires PDB to analyze properly 

• Stack-walking by PDB is very slow 

• Security 

• PDB must be not accessible from client / publisher 

• Performance 

• Must not slow down gameplay 
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JYP Prototype & PoC 


• FPS distribution 
with various tag filters 


Live Assist Performance Dashboard 




Recognize 


Prioritize 


Find Cause 


Fix 


FPS 



■ al 

SS 60 6S 70 75 

[^All 25-30 ■•'diff 

[^All 50-55 -7 diff 

axixiperatoK (f:\axdev\frainewoit(\ax<XMe\base\stnK|.cpp:1323) 

ax::RerxlerEflect:end(f:\axde\Afran>ewoit(\axsceoe\fx\ixeffect.cpp:1254) 

ax:K3xElemerYtHolder::getPtr (f:\axdev\frafnewo(KWcu«gxVgxeleinent 

ax::operator< (f:\axde\Aframewori(\axcore\base\stnng.cpp:1323) 
ax::DolMan::getltamFroiip(f:\axdev\fraiT)ewo(1(\axgameVdo\do(oan.h:40) 
ax::RxlModel::render (f:\axde\AfTamewoi1(\axscene\ix\nat)odel.q](>:168) 

ax::D3D9VertexBuffer::locl( (f\axdevVlrainewod(U»efide«\d3^ 

ax:;D3D9ShaderConstanfTable2::have (f:\axdev\fiainewori<\axiefKle 
ax:;RxModel;:rendef (f:\axdev\frainewoit(\axscene\i)(\ixmode<.q)p:16 

ax::D3D9Renderer:xlra\MndexedPrimitive(f:\axde\Afiatnewoi1(\axiendei\deviceVd3d9tendefef.cpp;149) 

ax:;D3D9lndexBuffiar;:locl( if:\axdev\ffamewori(\axiende<VI3d9\d3d9 


ax:KlxEletnetTt;:handleMouseEvent (f\axdevVftamewoifc\axuA(|x\gxe 

ax::D3D9ShadeiConstanfTable2::have(f;\axde\Aframewoi1<\axtende(\d3d9M3dxbaae.cpp:354) 

ax:;RenderEffect:etxl (f:Vaxde>AfrafT)ewoi1(\axscene\ix\fxeffect.cpp:12 

ax:: Sblng::- Stri ng (f:\axde\Aframewori(\axco<E\base\stnng qip: 1 58) 

k--(r\ j \r iv\ * \ u\. 1. 

ax::Dx ShaderValues::get (f:\axdev\fratnewoi1(\axrendei\dxVdxshadeivalue.q)p:370) 
ax::Dx StiaderValues::commit (f:\axde\AftBmewoik\axiende<VlxVdxshadeivalue.cpp:283) 

ax::D3D95hader:;setArrayValiie(f:\axdevMTainewo(k\axiefideiV13d9 


ax-RxMeshDalaBalcher-clearStaJa f\axdov\fiania5vodJa't'!reVXUxnreIdata ' 530 


** D*sha*derTl t(f3e^ta ori<\axrendeIdxWxshad ' hi 373)"^ ' 

PUG. M. 1 ..^LDD r\ J \r 1.' 

ax:: Stri ng::- Stii ng (f:\axde\AfiBmewo(l(\axcore\base\string.q]p: 1 58) 

ax::D3D9Shader::seMa(VWP(f\axdevMTameworiAaxrendetyJ3d9Wj3d93iiader.cpp:252) 

ax:KlxRoot::render (f:\axde\Afratnewod(\axuAgxVgxn>ot.q)p:328) 

ax:KjxElement:updateAction(f;\axde\AfraniewodAaxuAgx\gxelement.cpp:730) 

ax::D3D9Shader::setMa»WP (f:\axde\A6amewod(\ajaefidefVd3d9yi3 

ax::RxFieldFragment:TenderStiadow(f:\axdev\frame5voi1(\axgainectent\field\fxfieMfragmefit.cpp:208) 

ClientApp:aun (f:\axde\Aax\dlenndientapp.q)p:845) 

ax:: Stri ng:fformat (f:\axdev\frameworiAaxcore\base\sbing cpp: 1 045) 

ax::D3D9Renderer:4iresent (f:\axdev\framewoi1<\axrerdertdevjce\d3< 

ax::Dx StiaderValiies::commit (f:\axdev\framewod(\axrendeiVlx\dxshadeivalue.cpp:292) 


ax..RxGrannyMesh.aipdaleBB (f.\axdev\frame5vo(1(\axscene\rx\i)anes»i.cpp 1352) 
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JYP Prototype & PoC 



• FPS distribution 

with various tag filters 

• When clicked histogram, 
can compare call stack 
hot spots 


Live Assist Performance Dashboard 


Resolutiar:12&D*&00 


RfisokjtiMi 1680*1050 


1ield:Biizgaiter1 
field :Biizganiefi 1 1 @EHe 


field ;Biizganfer 1 @HeiD 
field :Bnzigaiter3@Elie 


field :Biizigaiter5@Eltie 
field :Biizgaiter7@EI(e 


field :Bnzgaiter9@EEte 
field Biizgafter Exam 


field Bfizganjefi SoerceiDomS _ 
field ;Housiig_001 


field Kobahotarll 


field :Kobal«atar1@£fte 


field :Kobal(jotai3 


field :KobaleaUi4@Efte 


field Kobalmtar? 


fi^ :Kobaloitar_EKain 


ax:»peralor< (f:\axdetAframewcn1AaKcone\base\string. cpp:1 323) 
ax::D'OMa ni-getJtemiFroijp ifAaxde'iAfra meworft\axg a meVdo\d om a n . h > 
ax:K5xElemeri(tHoldler::getPtr {fAaxdeV^framfewoiilt^axul'igjrlgxete 
ax::GxElernerit::hra ndleMouseEverit [f:\axdetAfra meworMaxu i^xlfgxe 
ax::D3D9VertexB jffer::lock {f:\axdeWrame\vor+rtax:FBndei\d3d9WJ.3d1 
ax::D3D9Shfl{lerConstantTable2::hiave (f:\axdev\framewojik\axiBinde 
ax::RxModlel ::rerMler ff:\axdev\fra mfi\vorfrtaxscene\cArxmQdel.cpp: 1 6. 
ax::□3□^H^ldlexBllffe^::lock {f:\axdev^.fram'ffivoTf<'-axnendei'd3d &W3d9- 
ax:K5xElennien(t::hra nrdleMouseEverit [f:\axde>Afra m■e^vc^r1^\axui^g>;\gxe■ 
ax::R■e^^dle^Effect:^e^Mi (f:\axdev^fra me\v(}r1t\axsoene''.DAixefr&i:t.cpp: 1 2 
ax:KjixElemen(t::ha ndHeMouseEvent iT:\axde>Afra mewortrtaxu iVgxIgxe 
ax::operal3or* {f:\axde>Aframewor1<\axcc}ne\math\matrix.cpp:1 OOS) 
ax::D3D9Sba{ter::selA rrayVal ue iTAaxdeV'.fra mewor1t\axrtndei\d 3d9' 
ax"RxMeBhDataBatclier:x:learStates iT:\axdev\framewc}r1AaxsceneV 
ax::DxShiadlerVa I ues "commit (f:\axdev\fra meiiVor1t).axrBndertdx\dxsh 
ax::RxGra nny Mes h:: jpdatseB B iT:'',axdeV'.fra meivorft'i.axsoene'tfx^Dcm.e 
ax::Strir»g::' String iTAaxdev',fram'ework\axc(}pe^J>ase\string.cpp:1 S3) 
ax::GxRoot:ren'dler (f:\axdev\frameworft\axu i'gx^gxraot.cpp: 32S) 
ax::D 3D9Sha'der::setMat5WVP iT:\axdev'.fra meworttaxMndeild 3d9\d3 
C I ientA pp::run (f:\axdev\ax\clienftclienta pp.cpp:S45) 
ax::D 3D9Rcndlerer::present tf:\axdev\framewc}rfrtaxiEnde['devlDe'd 3< 
ax:dietcllAt^Btd::less > iTAaxdev^frameivorlt^axconBVbaseVstlh ashmap.il 


ix:rfetchAt,std:H«ss > (f:\axde\Aframe\^ 


ax-RenderEffectKod ifAaxdeiAfra meworfAaxsoeneWw: 
ax:K5xEleii(iHeritHolder::'getPtr{fAaxdeV^fira.me\vorirtaxuf^ 
ax:xjperalDr< if:\axdev\frame^vorf?\axcoRe\f}aseVstring .qp 
ax::DoMa n::getitemFroup (f:\axdetAframfiwortAaxg ame^ 
ax::RxModlel ::rcr»dler {f:\axdev\framewoTi^axsceae\ixtai 
ax:K5xElement:hia ndleMouseEvent (f:\axdev\framfiwcH 
ax::D3D9Rendferer::draw1rMlexedlPrimitive (f:\axdev\fii 
ax:K5xElemien(t::hrandleMouseEverit (f:\axd eiiAfiramewoi 
ax::D3D9ShiaderConstanrt:Tablea::hiave (f:\axdeiiAfiramfi' 
ax::Stri ngii'^Stri mg if Aaxdev\framework\axc(}nB\basetetrii 
ax::operat3or* {fAaxdeiAframework'.axconelm ath\m atrix.q 
ax::DxSlTaderVa I ucb ::get iTAaxdeyAframewortAaxpender 
ax::DxShiaderVa I ucb "commit {f:\axdev\frameivtHlt\axre; 
ax:KjxElement::hia ndleMouBoEvent (f:\axdev\fraimewcH 
ax::RxMeB hrOataBatcheriKlea rStateB (f:\axdevAfraime\i 
ax::DxShiaderVa I ueB ::get (f:\axdeViifra meivorlAaxFendef 
ax::D3D9Shiader::setMatJ/WP (f:\axdev\firametvtHlk\axre 
axjzGxElennent^jpdateAction (f:\axdev\frameivEMlt\axuiP 
ax::RxFieldFragmerit::reniderSliadow{fAaxde\Afira.me^vi 
ax"Stri ngidormat if:\axde>Aframe\vorft\axcc>reVbase\strin' 
ax::DxShiaderVa I ucb iKommit {fAaxdetAframeivoiik^axrei 
ax"RxGra nrryMeB hi" jpdateB B (f:\axdev\framewcHtoaxi 


;RxGrannyMesh:n4>dateBB (r:\axde\AframewoikWx9ceneW\ixine9h.cpp;1352) 


T 










JYP Prototype & PoC 


• Can grouped by 

max frame interval as well 

• Represents 

laggy experiences well 
and can break down 
into call stacks and 
find causes 


Live Assist Perfonnance Dashboard 


Max Frame Interval 



QAII0-60ms‘<’diir 
ax::D3D9Renderef::dra»Hndexe<lPi 
ax::operatorc (f:\axdevViamewoik^a) 
ax::DoMan;:getltamFroup ifAaxdevM 
ax:K3xEletnentHolder::oetPtr (f:\axd 
ax::RenderEflect:xnd ifAaxdevVrait 
ax::D3D9ShadefConstanftable2: 
axiKBxElemeflCdtandleMouseEveni 
ax::D3D9Renderer:xlrawlndexedPi 


ix::RxModel::render (f:\ax(tev\franK 
ix:;RxMeshOataBalDher:x:learStal 
ix:;fetchAt,8td::les8 > (f:\axdev\fran 
ix;;operatdr* (r:\axde\AframeworiAax 
ix;;DxShadefValiie8::con«nit (f:\aw 
ix:K>xElen)em;diandleMou8«Evenl 
ix::D3D9Shader::8etMalVWP 
ix;:DxShaderValues::get ifAaxdevV 
ix;;String;:~ String (f:\axdev\framew< 
ix::DxShaderValues::get ifAaxdevV 
ix::D3D9Shader:»etArrayValiie if ' 
uctRxFieldFragmentrrenderShado 
ix::String;:format (f \axdev\framewo 
ix;;RxGrannyMesh:nipdateBB 
ix;:DxShaderValue8;x:ommlt If \ax< 
ix::RxGrannyMesh:nipdateBB 
ix;:RxModel:;render (fXaxdevMraitx 
>x:K3xPanel:yopagataEvent i 


All 50-100m8 •<din 
_tmainCRTStartup (f\dd\vdoolsVxt\atw32 
ClientApp:7un if \axdev\ax\dientVdientapp < 


ClientApp:diin if \axde'Aax\dientidientapp ( 
ax::D3D9VertexBuflen:lock if\axdevVrame 
ax:K3xElementHolder::getPtr (f\axde'Afiam 
axiKjxRooCTender (f \axdev\frame\vodAaxu 
ax::RenderEffect:»nd lf\axdev\frame\voriAi 
ax;:operator< lf\axde'Aftamewo(IAaxcore\b< 
ax::RxModel::rendef (f:\axdev\framewodAa) 
ax;:DoMan:;getltamFroup ifAaxde'Afiamewc 
ax:X3xElement;:handleMou8eEvent (f \axd< 
ax;:D3D9Renderef;:drawlndexedPrimitrve 
ax;:operator ifXaxdevVftamewodAaxcoreVtii 
ax::D3D9lndexBuffer::lock (f\axdevMrame\ 
ax:K3xElemen&:handleMou8eEvent if \axd< 
ax:KjxElement:handleMouseEvent if \axd< 
ax::D3D9ShaderCon8tanfTable2::have if\i 
ax::RxGrannyMe8h:dipdateBB if\axdevMra 
ax:delchAt,8td::le88 > if\axdev\fiamewo(kV 
ax::D3D9Shaden;8etMatWVP if\axdev\fran 
ax::DxShaderValue8;:conimit if\axdev\ftan 
ax:KjxPanel::propagateEvent if \axdev\fran 
axdSmng::' String (f\axdevMramewo(k\axco 
ax:£xElementnipdateAction if\axdevVian 
ClientApp:dvn IfAaxdeiAakidiennclientapp.c 
ax::String:rformat (f \axdev\framewod(\axcoi 


L>cJ All 100-150m8 'Tditr 

ClientApp:Tun (f \axde>AaxVdietihclienta|)p.c|)p;834) 

ClientApp;:run (f \axdev\ax\dient\dientapp.qip:866) 

ClientApp:;run (f\axde>AaxVdientNclientapp.c|>p:839) 

_tmainCRTStartup (fAdd\\xdoolsyittatw32Vllstumatexe.c:618) 

ax::D3D9RenderenxtravdndexedPrimitive (f \axdev\framewo«k\axrendei\deviceVd3d9rendefef.cpp: 1 S3) 
ax::D3D9Renderen:dravdndexedPrimitive (f \axdev\framewodAaxrendeiVlevice\d3d9iBndefer.cpp: 1 49) 
ax::operator< (f\axde'Aframewod<\axcoreV>ase\string.cpp:1323) 
ax:Ki(ElemenlHolder::getPlr(fAaxdev\framewoilAaxui'gx\gxelemert.h:460) 
ax:K3xElement::handleMou8eEvent(f\axde\Aframewoit<NaxukgxNgxelement.q>p:340) 
ax:;RxModel::rendef (f \axde'Aframewoik\axscene\ixVxmodel.cpp:168) 
ax;;RenderElfect:end (fAaxdev\framewodAaxsceneVxVxeffect.cpp:1254) 
ax;:DoMan;:getltamFroup|f\axdev\lraineworidaxgame\do\doman.h:40) 

ClientApp::run (f \axde>Aax\diennclientapp.cpp:845) 

ax;K3xElement::handleMou8eEvent(fAaxdev\framewodAaxukgx)gxelement.cpp:285) 
ax;:D3D9Shader:;8etArrayValue (f\axdevVrameworiAaxrende<Vj3d9id3d9sliadef.cpp:219) 
ax::D3D9Shader::8elMatVWP(f\axdev\framewoi1Aaxrende«\d3d9'd3d9sliader.cpp:252) 
ax::D3D9ShaderCon8tanfTable2::have(f:)axdev\framewoi1Aaxiende<V13d9)d3dxbase.cpp:354) 
ax;:DxShaderValue8;;get(f\axde\Aframewori(\axrendei\dxVlxsliade(value.cpp:370) 

ClientAppinun (fAaxde>AaxVciennclientapp.cpp:877) 

ax::operator (f\axde'AfrarneworiAaxcore\mathVnatiix.cpp:1008) 

ax::RxModel:a'ender (fAaxde\Aframewoik\axscene\ixVxmodel.cpp:212) 

ax::D3D9Renderer:g*-e8entif a- de/fMire., crt.i -lender de, -.e d3dsienderei cpp:588) 

ax:K3xElemenenv<lateAcfion(fAaxdev\framewodAaxukgx\gxelement.cpp:730) 

ax:: Slring::fonTiat (f \axde'Aframewo(k\axcore\base\string cpp: 1 039) 

AnimationBuildDirect(c:VdevelVadVgrannyVtVgranny_controled_animation.cpp:662) 

ax::BgLoader::Re80urce:nwaitLoad(f\axdev\framewofk\axca>e\system\t>gwod(er.q)p:635) 

ax:xyeratoc^ if:\axdev\framewo(k\axcore\l>ase\stnng.cpp:1288) 
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JYP Prototype & PoC 



• Can grouped by 

max frame interval as well 

• Represents 

laggy experiences well 
and can break down 
into call stacks and 
find causes 



Call stack hotspot in lag with 50~100ms: 
GxRootiRender (GUI) 
lndexBuffer::lock 

r ;:ill Qt;:irl<' hntQnnt in \^u with 1 nn~m<:' 

id (f:\axdev^fram 
tsl3nitTable2::li 

ax::D IJD &Vert&xB jffer::lock lT:\axdev^fra.me 

g V -nv Fliir¥io.F¥ri-lrtlHor"nort>tr if 'i avHa.-.,^fr:a rvi 

ax::0: 

lleMouseEvel 

ax:rfjxR' 00 t::renidler {f:\axd ev^fra mework^axu 

J::op 

drawl ndexedfl 



lleMouseEverit 

ax::operator< ff:\axdev^fra miewairk'iaxcore\da 

ax::Gj 


ax::RxModel::renidler (f:''.axdev'’.frameworK^',aj 

ax::Rj 

BgLoader::Resource::waitLoad 


cliefiKlearStat 

ax::DoMan::get]t3emFroijp {f:\axdev^frairr>ewc 

ax::Rt 


> {fAaxdev^ftarr 

ax::GxElemen(t::hia rKHeMouseEvemt (f:\axd'£ 

ax::DoMa n::ge'tJt]emFroup iT:\axdev^framewcMli^axg a me\d o\d oman . h :40 ) 


^v'.fraTTiCT/ork\ax 

ax::D !JD‘&Reniderer::draw1 mdexcdPri mitive 

Cl iemtA pp::rum iT:\axdeiAax^lient\clientapp.cpp:M5) 






lleMouseEveB 

ax::D IJD&I ndexB jffer::lock [f'.axde^Afra itict 

a::D 3tD9'Shadler::setA rrayVal ue iT:\axdev\framfiwork\axrendei'd 3d9'Vd 3d9sh ader.cpp:2 1 ■&) 


tMatWVP (f:\aS 


lx::D 5D^SIiader::setMalWVP (f \axdev\framewcn1AaxrendeAd 3d-9\d SdSsh ader.cpp:252) 


i::get [f:\axdev\f 

ax:K^xElemenrt::hia ndleMouseEvent (f:\axd€ 

ax::D;jD'&SlTiaderComBtan[tTablea::hiave {f\axdeVi,framew(}rt{\axrBnde[>d3d'&\d3dxbase.cpp:354) 


\axdev'.framew( 

ax::D3D‘&ShiaderConistanrt;Table3::hiave ff:\a 

ax::DxShaderVa 1 ues ::get iT:\axdev^frameivor1AaxnendeAdxVdxsh adervalue.cpp:370) 


i::get lT:\axdev^f 

ax::RxGraninyMeBhi::jpdateBB (f'.axdeiAfra 

C 1 iemtA pp::ium (f \axdeV^ax\dienficlientapp.cpp:S77) 


tArrayValue iT:'' 

ax:dietchiAt,s1il::less > lT:\axdel^^fralTmvcH1^^i 

ax::operator (f\axde^^framewc}rtAaxcc}ie\niath\m;atrix.cpp:1 OOS) 


CirenderShiada 

ax::D;jD‘&Shiader::setMatWVP (f:\axdev\fran 

ax::RxModel ::remder lT:\axdev^framewor1^^axsoe^e\DA[xm odel.cpp:21 2) 


axdeV'.framCT/o 

ax::DxShiaderVa 1 lies -commit (fAaxdev^fra IT 

ax::D 3D&Remderer::preBemt ff \axdev\fra me\vor1t\axrandei\deYiDe'vd 3d9ienderer.cpp: &BS) 


:updaleBB (f:\a. 

ax:GixPa t*bI ::propa-gateEvemt (f:\axdeV', fra rr 

ax::GxElen(*en(t::jpdatBAction if\axdeWrameiv(}r1<>,axui\gxVgxelem#nt.cpp:730) 


1 "commit [f:\axt 

ax::Stri mg::- Stri mg (f'.axdev^ftaiTveworl^axco 

ax::Strimg:rfornniat iT:\axdev\framew(}r1{\axc(ne\base\string.cpp:1(]3'9') 


lupdataBB if:\a. 

ax"'GxEl0n(ien[t:"ijpdal3eActiom {fAaxdev^frair 






r J 


■ (f:\axdeWrame 

C 1 iemtA pp::rjm iTAaxdetAax^tenfwilientapp.l 

I ax::BgLoa'der::ReBoijrce:a«aitLoad iT:\axdeiAframeivortoaxccne\systemVt}gw(}rker.cpp:63M 


lateEvemt (f:\axt 

ax::Stri mg:fcrmat ff \axdev\fram«work',axcor 
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Doing more to prevent risks 

• Major Stability Issues 

• Crashes 

• Memory Problems 

• Lag/Slowdown Issues 


• Doing more 
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Recog n i ze I 


Doing more in Runtime, On-demand 


• Often, need to put debug-log to find causes of not-reproducible problems 

• Usually takes 1 ~2 weeks to develop, test, and deploy for average projects 
running in multiple service regions 

• Isn't there a way to track state easily on-demand? 
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Live Watch technology 

• Telemetry on-demand 

• Not only for debugging 

• But also for Data Analysis, Hacking Prevention, Service Operation 

• Can retrieve variety of information that are not planned, prepared 

• Doesn't need to rebuild or redeploy 

• which takes days ~ weeks to release 

• Doesn't even need to add telemetry code 

• prevents log bloat 
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Live Watch technology 
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Recognize I Define I Find Cause I Fix 


“Blackbox" - a dashcam for a game 


• Records gameplay video constantly 

• only activated for developers / QA team 


• Ctrl+F1 2 directly pops up 
JIRA create-issue-window 

with recent gameplay video attached 

• Equipped with functionalities 

• auto-login 

• screenshots / edit to annotate 

• enables video preview with re-encoding 
(JIRA doesn't support, had to download) 
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Recognize ■ Prioritize 


n 


UserVoice" - hearing from user's voice 


m User Voice - £DIUI 0 1 


► aii.s^i 

► a ii.s^Ni (W/?ii^) 

► a 20. -OHSCf" 

► a 20. -OI^Q” 

► a3o.a#DiiA|7|g 

► a 40. aia 

► ^50. 

► a98.S7|llK?S0|S7|ll^/s 

► a 99. (1^012: 2i#) 

I 12. 


a22.^y\ 
a 30. 

i31.SS:^F 


a 33. 

a4o.2.^stf 
a 50.0 
asi. 

a 52. 


► S03. 

▼ a 11.7||?j2ha^ 

► a 3# 

► a 


2S| 


a iED|L||0|^!i7^ 2| 03-12 (S) 153 

[6i 


a y|^SSLK)| M«7|7^ [10] 03-12 (£) 235 

1* i4§Maa#°i bliitl ?1^Sii:l:^7^aa7| Aft-2 1,^; 


a^^§^e[5f05f0l!7|l4n 03-12(S) 34 

^^#017hO!U7|^ OH ¥|Lf otiSiy? 

□ l|°|!7|cH0] 03-12 (S) 0 

SOI |j7jLiw 2^ A\m^ ?]£ ~- 


□ 2i sg pE|7iE3oi — [0] 


03-12 (£) 0 


V iE□|^|0|A12r7^ y#0|7h7|m° [0] 03-12 (£) 0 

TllSOhOl^f V=e SS^I^I ^^mS-rr-rr 

> 7|5liE ^X\JL ^5^c^ [o; 03-12 (S) 0 

123 3« SO|7hO! iz|^ SOI 1?|- SOjblBI^ 31 3SW 




03-12 (S) 0 


I|7|fe2|| ojaiSL+M^a 

http://cafe.nave- :“i/nexond--“ nations/194422 44:gAt 
y* aS!7|| 0|7||Ai0t ttBI D7I214LI |j7|h||° 

“-!iaat $2 0 “ y\^ou 33|7|7| Dibiiais 52 oj^qi 

I 2 ^nim ## a| 0 | ^7|^q| ^s. 0 |[q| D3 S^o| 

1 q VS2! 7]£|0! lo^#7JS^^c|| 21 i insfl H^^aim 

n°A stioftAi ojgoeo 




2;j«25jU|^'53“,; 


03-10 
21 :25 

03-10 
21 -27 
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"UserVoice" 


Recognize B Prioritize I Find Cause I Fix 


hearing from user's voice 



2016 - 03-11 ® 2.^ 8:56 

_-^X-|M0|^ [uservoice] 

2016-03-11 ® (£D| Li|0|{^ZS) 


E _DominationsGM [gm-domi] 

0O| Qf\x\x\7[ SA|£|^ b^AlO| oiog 0|7|# #^of01 1 Met^X10|A1 D||A|X|# S^oiofAi A|S. 


gL|EHlj/^^gL|Eiij - 24A|^h 222H ^ ^At §X|, ^DH. ^OJ. ?j|S, Ohsll, 

ns. IS. n:7|, error. ^A||, uja, o|&1, :£=, a1|M, 



A|7( 

>£x^ ujiif 

3|fig iF^7^ 2J£5f^ 21£A| JI ... Q) M7| 

2016-03-11 04:18 

^^2101 OfsjAi 

1^£ £^£1A1 A^Ej|7^ 2i50|A|# MOi^SSAI^F 3....q a7| 

2016-03-11 02:38 

IS All^ 

o|S nSs “^*^^1-114 D|SE|»E| ois^q 7|^ C-) fi7| 

2016-03-11 00:19 

W1CU4 yx|^ 7|oi^ 7^x1l:f ^o\^x\ oi^om x|«0|^ . ...q M7| 

2016-03-10 23:11 

V^VLazenca save us / 

>Lazenca Save Us<(Jl *>.£« q^O| 0 |fl.AA@)^ ^ 22 I .. q_^ 

2016-03-10 22:22 

1 ^C>1A| 177H # El M7| 1 



2016-02-01 (1) 8:56 

_-^A]M0|r^_7|]^^ [uservoice_dev] 

2016-01-31 (^) ^AIMOI^ ^7h E?!!S (t0|Li|0|{^3) 


E _fl4El^ [uservoice.monitoring] 

O0| □|A|X|7^ SAIElt If^OI 0,0 2 0)7|# #=iSrq 1 MeT^X10|A1 Q|A|X|# SO|o>aia|£. 


in|M|0|^£S 


# 

201 6-01 -29(S) 

2016-01-30(S) 

201 6-01 -31 (^) 

1 

a|#2||o| 



2 

2!§ 

#A)7| 


3 

^¥11 


:2.2\ 

4 


7|# 

A^7^a| 

5 


hha| 

A^SE^^ 

6 

L^0| 


7|S 

7 

SOI 

SAj^r 

S^ 

8 


AjA|- 

1^ 

9 



aai:^ 

10 



2A|7| 

C| AM|o| M7| 



iD|L||0|A1^ 


# 

201 6-01 -29(S) 

2016-01 -30(E) 

201 6-01 -31 (^) 

1 


■y M a39 

yx^ 

2 

orai-AvLi 

ASO 

§SAi a24 

3 


7|o A49 

aa|^ am 

4 

# A, 

*A17| *,24 

S^ AW 

5 

aZLAi 

^31 

*33 
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Anomaly Monitoring 


NCS Ol^XIfi 

201613 02S 04S aao|fe^7;|fi 


2016i3 02S 04^ ^^0|^^7;|5 


1 . E! 5! “H nhOI EH : Al m - lomty SHET aiMlAi 

— HX»E.i-*ES 



. .f' ^ 


2016 - 01-07 (^) 9.^ 10;14 

RE : [ X | HO |^ Ehx | E |^ E ] 2015 ^ 12 fl 28 ^ - 03 ^ 
^t: I oiyoi [mot)']. I q 5 ! -ft [mk 3056 ] 

Ejoi MX| SSL] El S pivedata.monitorl; _DWl! [dw]: _E| 0| E1 “ 4j |j [da] 

23 . 0h:5.2hSH : ^ 7 [ (tT^) - NewUser UtHlAl g2l 

0m7Z^ NewUser • ^=T:| 

6,000 




7is 



0|^A|S 


^a(H) 


0|^>i7|.a_c 


SR 

NewUser 

5,189 

1,862 



24 .gseH®DiEi :a^(yx^)-uu m\M g?i 


< XIS ^ LH QB i ^^ 2 ^ ^ A 1 |Ej ^^ 0 | 2 ^ Si^LIQK 

aioia A | aOIAl M 2 I H 31 l^H Qff i ^^ 2 ^ HimthAI^I HheUQK 


^^^^ H ) 


30 

-11 
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Recognize I Prioritize I Find Cause 


Live Data Portal - Data Analysis On-demand 

• Variety of reports from live data as a service 
including: 

• Anomaly Reports 

• User Overlap Reports 

• Group Comparison 
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BottomMne 


• In live services, mitigating and preventing risks are very important 

• Prevent service not to stay unstable 

• Making problem visible is important 

In many cases, problems are not easily visible 

• If you make a solution or tool. 

Make a solution easy-to-use, on-demand for everyone, and 
Make a solution more reusable as you find duplicated costs 

• e.g. Crash Analysis, Data Analysis anytime for anyone 

• Minimizing time spent to recognize prevent risk is crucial 
Whatever it takes time, 

we can try to break it down to reduce time spent 


IIIV 



Thank You 


innover(cDnexon.co.kr 

https://twitter.com/_CKSong 


